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1.  INTRODUCTION 

Thermal  Infrared  (IR)  Images  of  the  ocean 
obtained  from  satellite  sensors  can  be  used  to 
study  ocean  dynamics.  Since  satellite  IR  images 
often  depict  mesoscale  features  clearly  (in  the 
absebce  of  cloud  cover  or  excessive  atmospheric 
water  vapor),  the  use  cf  AVHRR  Imagery  for 
various  oceanographic  applications  is  expanding 
rapidly.  The  extraction  of  mesoscale  ocean 
ferture  information  from  satellite  images 
usually  results  from  human  interpretation  of  the 
sea  surface  temperature  patterns  visible  in  the 
image.  With  the  proliferation  of  oceanographic 
analyses  that  utilize  satellite  data,  it  becomes 
highly  desirable  for  certain  applications  to 
move  from  labor-intensive  manual  interpretation 
of  satellite  Images,  toward  a  capability  for 
automated  interpretation  of  these  Images. 

Figure  1  shows  a  typical  IR  image  of  the 
Culf  Stream  region  of  the  Northwest  Atlantic. 
Images  such  as  Figure  1  are  the  primary  source 
of  information  leading  to  mesoscale  analysis 
products  like  that  shown  in  Figure  2.  The 
analysis  associated  with  the  generation  of 
Figure  2  was  performed  manually  by  analysts  at 
the  Naval  Eeistern  Oceanography  Center  in 
Norfolk,  Va.  The  long-range  objective  of  our 
work  is  to  develop  image  processing  and 
artificial  intelligence  algorithms  that  would 
allow  a  computer  to  produce  an  analysis  product 
such  as  Figure  2,  given  satellite  Imagery  such 
as  Figure  1.  This  is  a  difficult  image  analysis 
and  understanding  problem  due  to  a  lack  of 
precise  mathematical  descriptions  of  the  ocean 
features,  coupled  with  the  time-varying  nature 
of  these  features  and  the  complication  of 
frequent  partial  obscuration  by  cloud  cover. 

The  total  automation  of  this  task  is  .not 
presently  feasible,  but  efforts  toward  that  end 
will  result  in  many  tools  that  the  analyst  can 
use  to  reduce  labor  requirements  and  Improve  the 
quality  and  consistency  of  the  analysis. 

Several  previous  studies  have  addressed  the 
auto.matlon  of  the  analysis  of  IR  satellite 
imagery  for  mesoscale  features.  Gerson  and 
Gaborski  (1977),  Gerson  et  al.  (1982),  and 
Coulter  (1983)  have  Investigated  the  detection 
of  the  Gulf  Stream.  Janowltz  (1985)  and  Nichol 
(1987)  have  reported  some  success  in  detection 
of  eddies.  The  reader  will  find  additional 
Introductory  material  in  these  references. 

This  paper  describes  a  prototype  image 
analysis  system  thal  shows  rudimentary  skill  in 
automatically  locating  both  the  Gulf  Stream  and 


eddies  in  IR  images  of  the  Gulf  Stream  region. 
The  system  hEis  been  applied  to  twelve  test  cases 
resulting  in  some  preliminary  perform^mce 
statistics  that  are  presented.  Recommendations 
for  future  enhancements  are  also  given. 


2.  APPROACH 

The  automated  analysis  of  satellite  imagery 
has  been  formulated  conceptually  to  consist  of 
three  levels  as  shown  in  Figure  3.  The  lower 
level  involves  Inmige  segmentation  using 
conventional  Imsige  processing  techniques. 
Segmentation,  in  general,  can  be  edge-based  or 
region-based.  The  output  of  the  lower  level  is, 
therefore,  either  a  set  of  edges  or  a  set  of 
regions,  depending  on  the  segmentation  approach 
employed.  At  this  lower  level  'the  analysis  is 
focused  on  pixels.  Oceanographic  “knowlegde" 
does  not  enter  into  this  level.  The  interest  is 
simply  in  relationships  between  pixels  in  a 
local  neighborhood  that  signify  the  presence  of 
an  intensity  gradient  (i.e. ,  an  edge)  or  a 
region  of  uniform  Intensity. 

The  edges  or  regions  detected  in  the  low 
level  segmenter  are  then  passed  to  :  n 
intermediate  level  that  performs  the  twofold 
function  of  labeling  amd  feature  synthesis. 

First,  oceanographic  identities,  I.e.,  North 
Wall,  South  Wall,  warm  eddy,  cold  eddy,  etc.  are 
assigned  to. each  edge  or  region  created  by  the 
segmenter.  Edge  or  region  fragments  with 
identical  labels  are  then  collected 
(synthesized)  into  features  and  associated 
descrlpvive  parameters  such  as  position  emd 
radius  eu'e  calculated.  The  labeling*  function 
can  utilize  ancillary  data  collected  from  other 
sensors,  e.g. ,  an  altimeter,  as  well  as  other 
factors  such  as  continuity  from  the  previous 
emalysis,  oceanographic  context,  bathymetry, 
climatology  or  anything  else  that  might  help. 

The  labeling  and  feature  synthesis  functions  are 
envisioned  as  a  mixture  of  conventional  image 
processing  and  artificial  intelligence.  Some 
algorithms  may  be  pixel-based,  others  nay  be 
object  orelnted. 

A  third  analysis  level,  called  the 
oceanographic  expert  system  in  Figure  3, 
contains  a  higher  form  of  oceanographic 
knowledge.  The  data  structure  at  this  level  is 
object  oriented.  Data  sets  consist  of  lists  of 
object  types,  descriptive  parameters,  and 
coordinates.  Artificial  intelligence  is  the 
analysis  tool  applied  in  this  upper  level.  For 
example,  a  rule  base  describing  eddy  motions 
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Fig.  1.  Typical  satellite  IR  Image  of  the  Ho  <  'tream  region  of  the  North 
Atlantic.  Darker  shades  represent  warmer'  •  ,pv  :v  I'es  and  lighter  shades 
represent  cooler  temperatures.  A  spatially  variant  enhancement  has  been 
applied  to  this  image.  Correspondence  between  absolute  sea  surface 
temperature  and  gray  scale  has  been  lost. 


might  be  invoked  at  this  level  to  move  known 
eddies  about  in  the  analysis  product  during 
periods  of  cloud  cover,,  when  direct  observation 
of  the  eddy  is  Impossible 

3.  SEMI -AUTOMATED  MESOSCALE  ANALYSIS  SYSTEM 

The  general  approach  outlined  in  the 
previous  section  has  been  translated  into  a 
specific  implementation  called  the 
Semi -Automated  Mesoscale  Analysis  System 
(SAMAS).  Version  1.0  of  SAMAS  is  shown  in 
Figure  4.  The  components  of  Figure  4  are 
briefly  described  below. 

3. 1  Segmentation 

An  edge-based  image  segmentation  scheme  has 
been  chosen  for  SAMAS  Version  1.0.  The  edge 
detection  algorthm,  developed  specifically  for 
this  application  from  image  texture  theory,  has 
been  described  in  Holyer  and  Peckinpaugh  (1989). 
Cayula  and  Cornilllon  (1990)  have  also  developed 
a  special  edge  detection  module  for  IR  satellite 
Imagery  of  the  ocean  Cambridge  et  al.  (1990) 
are  Investigating  the  incorporation  of  both  edge 


and  region  base!  an,,  caches  into  an 
oceanographic  image  segmonter.  Clearly,,  there 
are  several  options  available  for  oceanographic 
image  segmentation.  Version  1  0  evaluated  here 
utilized  the  original  Holyer  and  Peckii  paugh 
(1989)  edge  detector.  As  a  result  of  Ihr  oiher 
work  underway  in  this  area,  future  versions  of 
SAMAS  will  undoubtedly  benefit  from  improved 
segmentation 

3  2  Feature  Labeling 

SAMAS  featuie  labeling  is  performed  by 
nonlinear  probabilistic  relaxation  (Krishnakumar 
et  al.,  1990).  Relaxation  labeling  requires  a 
first  guess  of  the  probabilities  of  edge 
segments  belonging  to  each  of  the  object  types. 
Tills  first  guess  is  noi-mally  piovided  by  taking 
the  analysis  product  from  the  p-evicus  analysis 
period  and  moving  the  features  lorward  in  time 
to  the  present  analysis  date.  Moving  features 
ahead  in  time  is  accomplished  by  the 
oceanographic  expert  system  shown  at  the  right 
e-'ge  of  Figure  4.  Ititialization  with  the 
previous  analysis  weights  temporal  continuity 
heavily  as  a  factor  Influencing  feature 
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labeling.  This  weighting  Is  considered 
desirable  since  temporal  continuity  over  scales 
of  days  is  a  major  characteristic  of  mesoscale 
ocean  features.  Initial  oceanographic  label 
probabilities  for  each  edge  segment  are  then 
iteratively  adjusted  until  they  converge  to 
values  representing  minimum  uncertainty  (or 
maximum  consistency)  in  the  assignment  of  one  of 
the  object  labels  to  each  edge  pixel. 

Relaxation  labeling  has  been  applied  to  a 
variety  of  Image  processing  problems,  such  as 
pixel  classification,  that  are  related  to  this 
work  (e.g. ,  Eklundh  et  al.,  1980  and  Davis  et 
aj.,  1983).  The  relaxation  labeling  aporoach  is 
elegantly  described  in  Rosenfeld  et  al.  (1976). 

Genetic  algorithms  are  under  investigation 
as  an  alternate  technique  ‘'or  labeling 
oceanographic  features.  Ankenbrandt  et  al. 

(1989)  describe  the  genetic  algorithm  approach 
to  feature  labeling. 

3.3  Feature  Svntheg'g 

Labeled  edges  provide  fragmented 
representations  of  oceemographlc  features.  For 
example,  partial  cloud  cover  may  result  In 
detection  of  only  a  few  Isolated  pieces  of  the 
North  Wall.  Likewise,  the  complex  spiral 
structures  associated  with  eddies  can  lead  to 
the  association  of  numerous  small  edge  fragments 
with  an  eddy.  Constructing  a  continuous  Gulf 
Stream  or  a  complete  circular  eddy  from 
fragmented  renditions  of  these  features  is  here 
called  feature  synthesis. 


Molinelli  and  Flannigan  (1987)  have 
developed  a  method  of  constructing  a  continuous 
Gulf  Stream  from  freigmented  segments.  Their 
method,  which  is  Incorporated  into  Version  1.0 
of  SAMAS,  is  based  on  a  weighted  sum  of  Gulf 
Streaun  normal  modes.  Normal  modes  are  the 
eigenvectors  of  a  covariance  matrix  describing 
Gulf  Stream  shapes.  The  normal  mode  weights 
required  to  produce  a  least-squares  minimization 
of  distance  between  the  Gulf  Stream  edge 
segments  auid  the  resulting  normal  reconstruction 
are  found  by  an  iterative  least-squ-  •'es  method. 
Gulf  Streaun  shapes  synthesized  fr^.n  iragments  in 
this  manner  au-e  continuous  and  carry  through 
au'eas  of  could  cover,  giving  an  estimate  of  Gulf 
Stream  location  amd  shape  beneath  the  clouds. 

Wau'm  amd  cold  eddirs  are  synthesized  using 
the  circular  Hough  transform  (Duda  amd  Hart, 
1972).  In  general  the  hough  transform  consists 
of  remapping  edge  points  from  image  space  into  a 
parameter  space.  The  Hough  transform  for 
circles  projects  edge  points  into  a 
3-dimenslonal  parauneter  space  (the  pairameters 
being  x-posltion,  y-poslticn,  and  radius).  The 
parameter  space  is  quamtized  into  discrete  bins 
and  is  known  ar  the  "accumulator  array".  Each 
edge  point  in  the  image  that  has  been  labeled  as 
eddy  "votes"  for  all  those  circles  of  which  it 
may  be  a  part  by  incrementing  the  appropriate 
accumulators.  Local  maxima  in  the  accumulator 
array  correspond  to  the  circular  shapes  within 
the  edge  image.  This  method  will  find  the  most 
prominent  eddies  and  give  (x,  y,  r)  parauneter 
sets  which  permit  filling  in  the  eddies  even 
when  slgnificamt  parts  are  missing. 
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SATELLITE  IMAGE 


OTHER  SENSORS 
ANCILLARY  DATA 


Fig.  3.  A  three-tiered  approach  to  automated 
oceanographic  satellite  Image  analysis. 


Editor  modules  also  provide  record  keeping 
functions  by  tagging  each  feature  with  posltloj 
and  size  parameters,  and  with  ancillary 
Information  such  as  the  source  of  the  feature, 
e.g. ,  automatically  synthesized,  propaguted 
forward  In  time  from  the  previous  analysis  via 
the  expert  system,  or  manually  entered  by  the 
user. 


3-5  Oceanographic  Expert  S-/stem 

The  oceanographic  expert  system  consists  of 
rules  about  time  evolution  of  mesoscale  features 
In  the  Gulf  Stream  region.  Eddy  motion,  changes 
In  eddy  size  with  time.  Interaction  between 
eddies  and  the  Gulf  Stream,  and  Gulf  Stream 
phase  velocities  are  factors  covered  by  this 
rule-base  (Lybanon  et  aJ.,  1986).  Performance 
of  this  rule  base  In  describing  eddy  motion  has 
been  evaluated  by  comparison  with  manually 
Interpreted  satellite  Inmigery  Eddy  position 
forecasts  out  to  7  days  have  been  shown  to  be 
better  than  the  asstimptlon  of  no  notion  In  63J4 
of  the  70  cases  studied.  Accuracy  of  the  Gulf 
Stream  dynamics  In  the  expert  system  has  not  yet 
been  evaluated. 

The  function  of  the  expert  system  In  SAMAS 
Is  twofold.  First,  It  forecasts  feature 
positions  so  that  these  dynamic  features  can  be 
plotted  In  their  approximate  locations  during 
Ijfeplods  of  cloud  cover,  where  direct  observation 
Is  not  possible.  Second,  the  feature  positions 
from  the  previous  analysis  are  adjusted  to 
provide  a  better  first  guess  for  relaxation 
labeling  in  the  present  analysis. 


Fig.  4.  A  functional  block  diagram  of  SAMAS 
Version  1.0  automated  '.sage  analysis  system. 


The  clsxular  Hough  transform  has  been  used 
In  a  variety  of  appl tcatlons  requiring  the 
detection  of  circular  features  In  Imeiges.  For 
example.  Cross  (1988)  applies  It  to  find 
circular  geologocal  features  In  Landsat  Imagery; 
Klmme  et  el.  (1975)  apply  It  to  the  detection  of 
tumors  In  chest  radiographs. 

3.4  Interactive  Editors 

SAMAS  provides  Interewtlve  software  modules 
that  allow  the  analyst  to  view  the  results  and 
to  delete  or  modify  features  that  were 
automatically  synthesized.  Features  that  were 
missed  by  the  automated  techniques  can  be 
manually  added  by  using  the  Interactive  editors. 


4.  SAMAS  PERFORMANCE  EVALUATION 

Twelve  test  cases  from  April  and  May  of 
1989  were  selected  for  evaluation  of  the 
automated  feature  location  perforaance  of  SAMAS. 
The  data  set  contained  36  Images.  One  test  case 
consisted  of  anywhere  from  2  to  5  of  these 
Images  mosalced  together  by  maximum  temperature 
techniques,  to  give  a  single  test  Image. 

Maximum  temperature  compositing  was  performed  to 
minimize  cloud  cover  In  the  test  images.  Figure 
5  shows  the  test  case  Image  that  is  the  most 
cloud  free  of  the  twelve.  All  cases  will  be 
shown  In  the  poster  session. 

The  twelve  test  caises  were  processed  using 
the  algorithms  described  In  Section  3.  The 
result  w^ls  an  automatically  generated  mesoscale 
emalysis  for  each  case.  The  automated  analysis 
of  the  test  Image  shown  In  Figure  5  Is  given  in 
Figure  6.  Note  that  Figure  6  was  produced 
totally  without  human  interpretive  Input,  i.e. , 
the  interavtive  editors  available  in  SAMAS  were 
not  used  in  this  case.  Positions  of  the 
features  seen  In  Figure  6  and  in  the  other 
eleven  automatically  generated  analysis  products 
were  compared  with  feature  positions  determined 
from  manual  Interpretation  of  these  images  in 
conjunction  with  the  GEOSAT  Ocean  Applications 
Pi’ogram  (GOAP)  (Lybanon  et  al.,  1990). 

Preliminary  comparisons  show,  for  example, 
that  the  prominent  warm  eddy  to  the  north  of  the 
Gulf  Stream  in  Figure  5  was  detected 
automatically  In  90K  of  the  cases  where  the 
human  analyst  was  able  to  locate  It.  The  mean 
distance  between  centers  of  this  eddy,  as 
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Fig.  5  A  test  image  used  to  evaluate  SAHAS  Version  1.0.  Enhancement  is  the 
same  as  in  Figure  1. 


determined  meinually  and  automatically,  was  27 
km.  The  automatic  algorithms  assigned  a  radius 
value  to  this  eddy  that  vras,  on  the  average,  30% 
larger  than  the  radius  assigned  by  the  human 
Interpreter.  Additional  accuracy  results  for 
cold  eddies  and  for  Gulf  Stream  position  will  be 
presented  on  the  poster. 

5  CONCLUSIONS 

The  feasibility  of  automated  analysis  of 
satellite  Imagery  for  p.-sltlons  of  mesoscale 
ocean  features  has  been  demonstrared. 

WecJcnesses  in  Version  1  0  have  been  Identified 
and  now  form  the  rational  for  fvirther  research. 
Specifically,  Improvements  have  been  initiated 
In  the  au-ea  of  segmentation,  where  a  new 
segmenter  utilizing  both  edge  and  region 
Information,  Is  under  development  (Cambridge  ct 
al.,  1990).  This  first  evaluation  has  also 
revealed  that  the  feature  labeling  algorithm  Is 
too  closely  tied  to  the  Initial  guess. 
Improvements  In  this  area  are  also  underway 
(KrishnEdtumeLT  et  al.,  1990).  A  new  approach  to 
feature  labeling  using  genetic  algorithms  Is 
also  under  Investigation  ( Ankenb''andt  et  al., 
1989). 


Fig.  6.  Automated  analysis  of  Figure  5  showing 
the  Gulf  Stream  position,  one  warm  eddy,  Euid  two 
cold  eddies. 
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